Skip to content

fix(sdk,api,local): surface executions interrupted by a daemon restart - #1888

Open
ra-co88 wants to merge 1 commit into
UsefulSoftwareCo:mainfrom
ra-co88:fix/execution-tombstones
Open

fix(sdk,api,local): surface executions interrupted by a daemon restart#1888
ra-co88 wants to merge 1 commit into
UsefulSoftwareCo:mainfrom
ra-co88:fix/execution-tombstones

Conversation

@ra-co88

@ra-co88 ra-co88 commented Aug 30, 2026

Copy link
Copy Markdown

What

Executions interrupted by a daemon restart are now recorded as tombstones and surfaced through the API and SDK as interrupted, instead of lingering forever in a running state.

Why

When the daemon dies mid-execution, in-flight executions had no terminal state — they read as running indefinitely, breaking retries, UX, and any reconciliation logic. The record store now writes tombstones on startup for executions that belonged to the previous process, and the API/SDK expose the interrupted state.

What changed

  • makeExecutionRecordStore writes tombstones for prior-process executions at boot.
  • The executions API and SDK executor surface the interrupted state.
  • The local app wires the store.

Test plan

  • Record-store suite: tombstone written once, idempotent re-sweep, boundary reads.
  • Handler suite: interrupted executions return the terminal state, not running.

6 + 4 tests green against current main.

@ra-co88

ra-co88 commented Aug 30, 2026

Copy link
Copy Markdown
Author

Heads-up on the red E2E (cloud 13of16) check here: it's failing on main itself (e.g. the Version Packages runs), so it's pre-existing rather than from this PR. It's the cap-eviction scenario tripping over workerd resetting session Durable Objects mid-initialize when the test opens its burst of sessions — diagnosis and a proposed fix in #1895.

@ra-co88 ra-co88 left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: approve — daemon-restart tombstones, correctly designed and tested. (Comment review: GitHub blocks formal self-approval on your own PR.)

Supersession check (upstream/main @ 2dc399e): packages/core/sdk/src/execution-records.ts does not exist upstream, the tombstone handler test does not exist upstream, and the "interrupted" markers in upstream's executor.ts are unrelated comments about fiber interruption. Not superseded — interrupted executions still linger as running upstream.

The design is the honest shape:

  • Tombstones at the storage layer: makeExecutionRecordStore writes a boot-time sweep (sweepInterrupted) that marks every non-terminal record from the previous process as interrupted. The sweep is idempotent and reads a durable live-index that terminal writes consume — the mechanism is documented in-file.
  • The API surfaces the truth as 404 + InterruptedExecutionError, not a fake terminal state: the class doc is explicit that an interrupted execution means "nothing ran, re-triggering is safe" — the right semantic for retries (idempotent re-trigger, never double-run).
  • Boundary discipline: completed tombstones are immutable (test: "completed tombstones do not resurrect"), stale paused tombstones still surface interrupted (sweep-miss tolerance), and the no-tombstone path falls through to approval-expired — each boundary has a dedicated negative test.
  • Secrets never enter the tombstone (spec'd in-file: "no secret leakage").
  • Pause records are tombstoned best-effort in the handler, so a restart mid-pause also resolves rather than hangs.

Negative-control discipline holds: 4 handler tests + 6 SDK tests, all asserting boundary behavior (stale, missing, already-terminal, idempotent re-sweep), not just the happy path.

CI caveat, not blocking: the single failing shard (E2E cloud 13of16, run 2026-08-30) is the cap-eviction openSession: no mcp-session-id flake — the exact transient #1895's restart-envelope retry fixes; this branch predates it and touches no session code. A rebase onto current main picks up the backstop.

Good to merge after rebase.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants